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In "How Is the Afterschool Field Defining Program 
Quality?" in the fall 2009 issue of Afterschool Matters, 
Palmer, Anderson, and Sabatelli review recent research 
on quality frameworks. They conclude that six domains 
of quality are especially critical: supportive relationships, 
intentional programming, strong community partner¬ 
ships, promotion of youth engagement, physical health 

and safety, and continuous quality improvement. This 
review and other recent cross-program or meta-analytic 
efforts to identify core components of quality after¬ 
school programs (Durlak & Weissberg, 2007; Metz, 
Goldsmith, & Arbreton, 2008) provide a valuable op¬ 
portunity for afterschool providers to reflect on their 
practices. In addition to correlational assessments of 
program attributes and outcomes, however, the field 
also needs data about how specific interventions have 
improved quality in afterschool programs (Granger, 
Durlak, Yohalem, & Reisner, 2007; Pittman, Smith, & 
Finn, 2008). 


This paper describes one approach to such quality 
improvement efforts: the Quality Improvement System 
(QIS) implemented by Prime Time Palm Beach County 
(Prime Time) in Palm Beach County, Florida. Prime 
Times QIS is recognized as one promising systemic ef- 
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PALM BEACH COUNTY'S FIVE AFTERSCHOOL 
QUALITY STANDARDS 


STANDARD ONE 

Administration, Program Organization, Procedures, and Policies 

Provide Solid Framework for Afterschool Program 

• The afterschool program is structured and organized to ensure the 
health and safety of children and youth in the program. 

STANDARD TWO 

Supportive Ongoing Relationships between and 

among Youth and Staff 

• The afterschool program staff involves youth as partners in the 
program and encourages children and youth to work together. 

STANDARD THREE 

Positive and Inclusive Environment for Youth 

• The afterschool program staff creates an environment that allows 
children and youth to feel a sense of belonging. Staff sets clear limits 
and assists youth in managing conflict. The afterschool program is 
equipped to provide a range of activities from which youth can choose. 

STANDARD FOUR 

Youth Development and Challenging Learning Experiences 

• The afterschool staff is trained in strategies that result in providing 
children and youth with positive learning experiences. 

STANDARD FIVE 

Outreach to and Activities for Families 

• The afterschool program supports family involvement by helping 
parents connect with their child's education and fosters positive 
interaction among families. 


fort to improve quality in the afterschool field (Yohalem, 
Granger, & Pittman, 2009). As a systemic effort, the QIS 
incorporates many elements of quality improvement be¬ 
ing implemented in other afterschool settings, including 
assessment, planning, coaching, training, and network¬ 
ing. The experience of developing and implementing 
QIS offers opportunities for learning at multiple levels: 
how a systemic response can develop from an initial 
concern about low quality; how afterschool program di¬ 
rectors experience the introduction and continuation of 
such a systemic approach; and how a system’s multiple 
strategies can be refined and aligned. This experience 
also provides an opportunity to step back and reflect on 
the relative importance of specific contributing elements 
in the larger system. 

Our primary intention in this article is to docu¬ 
ment findings and lessons from a systemic quality ap¬ 
proach in a way that can inform the crucial discussion 


of quality in the afterschool field and its 
implementation in afterschool programs 
(Yohalem et al., 2009). In addition to de¬ 
scribing the implementation of the QIS, 
this article describes its effects on pro¬ 
gram quality. Following a description of 
the QIS, we review findings from an eval¬ 
uation of afterschool programs in the QIS 
conducted by the David P. Weikart Center 
for Youth Program Quality. This evalua¬ 
tion demonstrates the specific value of 
QIS over time for participating organiza¬ 
tions, comparing QIS results to those of 
afterschool programs outside Palm Beach 
County. We also summarize key findings 
from an independent longitudinal pro¬ 
cess evaluation of QIS conducted by 
Chapin Hall at the University of Chicago 
during the past four years. This evalua¬ 
tion allows us to “look inside the black 
box” to describe and analyze the ways in 
which the system has supported quality. 
We conclude by arguing that the long¬ 
term and iterative process of the QIS has 
been essential to creating a community 
of afterschool providers who value high- 
quality programming. 

Palm Beach County's Quality 
Improvement System 

Prime Time Palm Beach County is a non¬ 
profit afterschool intermediary organiza¬ 
tion. Its framework was developed a decade ago by a 
county-wide consortium in response to concerns about 
low-quality afterschool programs. Since its inception, 
Prime Time has spearheaded efforts to create standards, 
supports, and resources for Palm Beach County after¬ 
school providers. As part of this emphasis, Prime Time 
coordinated a work group of key stakeholders that cre¬ 
ated Palm Beach County’s five quality standards (see 
box), which predate but overlap with the recently devel¬ 
oping consensus in the field about key aspects of quality 
(Durlak & Weissberg, 2007; Miller, 2005; Pittman et al., 
2008). 

These standards were initially intended to be 
benchmarks for an afterschool Quality Rating System 
(QRS) similar to one already in place for early educa¬ 
tion childcare centers. The QRS was designed to assign 
star ratings and provide incentives to early care pro¬ 
grams meeting quality thresholds. But as Prime Time 
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continued to clarify its core vision for quality improve¬ 
ment, the work group reoriented its philosophy and 
renamed the initiative the Quality Improvement System, 
an approach that since then has demonstrated a record 
of improving and sustaining quality for large numbers 
of afterschool providers. 

The QIS is a low-stakes approach designed to help 
afterschool programs continuously improve the quality 
of their services. Instead of using a system of rewards or 
sanctions, the QIS provides training and a quality as¬ 
sessment tool that give afterschool providers a ground¬ 
ing in the five quality standards. It also offers support 
and resources to help providers work toward the stan¬ 
dards. The process begins with an initial external base¬ 
line assessment. Then the afterschool program is as¬ 
signed a quality advisor, a Prime Time employee with 
expertise in youth development, afterschool program¬ 
ming, and coaching. The quality advisor and program 
leaders use the assessment to create an improvement 
plan, which guides the programs subsequent quality 
improvement efforts. This improvement plan includes 
recommendations for training or other supports and re¬ 
sources needed to improve quality. Programs also con¬ 
duct a self-assessment to assess their quality and to rein¬ 
force their understanding of the quality standards. This 
cycle of assessments and supports is expected to be long¬ 
term; participating programs receive an annual external 
assessment with new plans, specific suggestions for im¬ 
provement, and links to additional resources and tech¬ 
nical assistance. 

The Program Quality Assessment 

The quality assessment tool used in the QIS is a modi¬ 
fied version of the HighScope Youth Program Quality 
Assessment (YPQA, HighScope Educational Research 
Foundation, 2005b), which was chosen after careful 
consideration of various out-of-school time quality as¬ 
sessments. To align the existing assessment tool and 
Prime Time’s five quality standards, HighScope Youth 
Development Group (now the David P. Weikart Center 
for Youth Program Quality) was contracted to adapt the 
tool for local use. The result of this work is the Palm 
Beach County Program Quality Assessment (PBC-PQA, 
HighScope Educational Research Foundation, 2005a), 
which consists of two major parts, Form A and Form B. 
Form A is a point-of-service observational tool that con¬ 
sists of four domains: Safe Environment, Supportive 
Environment, Interaction, and Engagement. Each do¬ 
main includes subcategories (scales) with specific indi¬ 
cators (items). Prime Time contracts with a local organi¬ 


zation that trains and supervises external assessors to 
use the PBC-PQA to conduct the observation of three 
staff members at each afterschool site. An afterschool 
program’s Form A score comprises the average of these 
three observation scores. 

Form B was designed to capture program quality at 
the organizational level using four domains—Youth- 
Centered Policies and Practices, High Expectations for 
Youth and Staff, Organizational Fogistics, and Family— 
and their corresponding scales and items. The informa¬ 
tion for Form B is collected through interviews with ad¬ 
ministrative staff and reviews of program documents. 
We include findings from both Form A and Form B as¬ 
sessments below. 

QIS Outcomes 

As part of its work in Palm Beach County, the Weikart 
Center analyzed assessment data that began with the 37 
school- and community-based afterschool sites that par¬ 
ticipated in the 18-month QIS pilot project in 2006- 
2007 and continued through assessments of the 90 sites 
participating in 2008-2009. Simultaneously, the re¬ 
searchers were collecting YPQA assessment data in three 
states from organizations serving elementary school-age 
children; these providers met basic organizational and 
program criteria including having full-time administra¬ 
tors, delivering year-round programming, and produc¬ 
ing a weekly schedule of offerings. This rich data set al¬ 
lowed the Weikart Center to estimate the impact of the 
QIS intervention in three ways: 

• By measuring changes in scores within a single pro¬ 
gram year during which a program improvement plan 
was being implemented 

• By measuring changes in scores over two or more years 

• By comparing scores of programs in QIS to those of 
similar programs using the YPQA outside of Palm 
Beach County 

Changes in One Program Year 

As indicated in Table 1, the Weikart Center researchers 
concluded that, with the exception of one domain—Sup¬ 
portive Environment, which was relatively high at base¬ 
line—assessment scores increased during the 2008-2009 
academic year. The Center determined that, although the 
differences were not statistically significant, “the fact that 
measured quality is higher later in the year suggests that 
the QIS, with its mix of assessment, training and techni¬ 
cal assistance, is working” (Sugar, Pearson, Smith, & 
Devaney, 2009, p. 2). 
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Table 1. 2008-2009 Palm Beach County Mean Scores across Time, Form A 



T1 

(N = 99 offerings) 

T2 

(N = 74 offerings) 

T3 

(N = 97 offerings) 


9/1/08- 11/30/08 

12/1/08-1/31/09 

2/1/09-4/31/09 

Safe Environment 

4.90 

4.92 

4.95 

Supportive Environment 

4.43 

4.24 

4.35 

Interaction 

3.57 

3.60 

4.62 

Engagement 

2.80 

2.82 

2.91 


Adapted from Sugar et al, 2009 

Note: Statistical significance for difference of means was tested across time points. There were no statistically significant differences. 


Table 2. Comparison of Pilot, Baseline, and Year 2 Domain Scores, Form A 



Pilot Baseline 
Mean 

(N = 23 sites) 

Pilot Reassessment 
Mean 

(N = 23 sites) 

2008 Baseline 

Mean 

(N = 24 sites)* 

2009 Reassessment 
Mean 

(N = 24 sites)* 

Safe 

Environment 

4.39 

4.75 

4.75 

4.85 abc 

Supportive 

Environment 

3.91 

4.25 

4.26 

4.37 a 

Interaction 

3.26 

3.51 

3.43 

3.65 ac 

Engagement 

2.53 

2.81 

2.83 

3.00 a 


Adapted from Sugar et al, 2009 


Table 3: Comparison of Pilot, Baseline, and Year 2 Domain Scores, Form B 



Pilot Baseline 
Mean 

(N = 23 sites) 

Pilot Reassessment 
Mean 

(N = 23 sites) 

2008 Baseline 

Mean 

(N = 24 sites)* 

2009 Reassessment 
Mean 

(N = 24 sites)* 

Youth-Centered 

Policies and 

Practices 

2.89 

3.41 

4.15 

4.46 abc 

High Expectations 
for Youth and Staff 

3.99 

3.56 

4.80 

4.82 ab 

Organizational 

Logistics 

4.33 

4.73 

4.20 

4.20 b 

Family 

3.74 

4.23 

4.54 

4 79 abc 


Adapted from Sugar et al, 2009 


‘Between the pilot and baseline years, one of the pilot sites split into two sites. 

a Indicates significant difference between pilot baseline mean and 2009 reassessment mean at p < .05 level. 
b Indicates significant difference between pilot reassessment mean and 2009 reassessment mean at p < .05 level. 
c Indicates significant difference between 2008 baseline mean and 2009 reassessment mean at p < .05 level. 
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Figure 1. Palm Beach County domain scores vs. large reference 
sample of programs serving elementary-age youth 
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From Sugar et al, 2009 

Note: Statistical significance for difference of means was tested between the two 
samples. All differences between the domain means were statistically significant 
at the p < .05 level. 


Changes across Program Years 

In addition to changes within a single year, 
participating organizations showed robust 
improvement in both Form A and Form B 
scores across multiple years. As indicated in 
Table 2, with the exception of one data point 
(2008 Interaction score), the aggregate scores 
of each domain on Form A stayed the same or 
increased from one year to the next. Weikart 
researchers suggested that “gains produced 
by the QIS intervention are both stable and 
sustainable” (Sugar et al., 2009, p. 7). 

Form B scores, provided in Table 3, fol¬ 
low a similar trajectory. With the exception of 
the Organizational Logistics domain, increases 
in these scores over time are both consistent 
and statistically significant. This pattern sug¬ 
gested to the Weikart researchers that “core 
components of the QIS are being successfully 
institutionalized in management policies and 
practices” (Sugar et al., 2009, p. 10). 

Comparisons to a Larger Sample 

Quality scores of afterschool programs serving elemen¬ 
tary school-age children in other states offer another way 
to assess the effect of the QIS. As Figure 1 indicates, Palm 
Beach County QIS programs scored higher in all domains 
of the PBC-PQA observational tool, Form A, than peer 
organizations using the comparable YPQA tool on which 
the PBC-PQA was based (Sugar et al., 2009). The differ¬ 
ences between QIS programs and comparison programs 
were statistically significant in all cases. 

Together, these analyses provide specific measures of 
the improvements in program quality experienced by 
agencies participating in the QIS. 

What Makes QIS Effective 

Key findings from an overview of annual process evalua¬ 
tions, conducted over the same time by Chapin Hall at 
the University of Chicago (Baker, Spielberger, Lockaby, 
& Guterman, 2010; Spielberger & Lockaby, 2006; 
Spielberger & Lockaby, 2007; Spielberger, Lockaby, 
Mayers, & Guterman, 2008; Spielberger, Lockaby, 
Mayers, & Guterman, 2009), provide additional detail 
on the operations and effects of the QIS. 

Chapin Hall researchers observed the development 
of supports for quality in afterschool programs in Palm 
Beach County between 2004 and 2009. They conducted 
more than 50 cross-sectional and longitudinal interviews 
with program and agency directors, interviews with 


Prime Time staff, and observations at planning meetings 
and other events. They also reviewed program documen¬ 
tation of assessment scores and use of QIS services. This 
research identified several aspects of the QIS that were 
important in explaining rising trends in program quality 
among participating agencies. 

Low-stakes Support 

A key aspect of the QIS is its low-stakes and supportive 
approach. The switch in name from a Quality Rating 
System to the Quality Improvement System was accom¬ 
panied by related conceptual changes and practices. 
Instead of rewarding agencies with star ratings and in¬ 
centives only when quality thresholds had been met, the 
QIS front-loaded its support. Incentives of $1,500 to 
$3,500, based on enrollment, were provided to agencies 
as they joined the QIS. Instead of using only external as¬ 
sessors, program staff were also trained in self-assessment 
so that both external and internal assessments using the 
PBC-PQA tool would be available to describe program 
strengths and weaknesses. 

Framing quality improvement as a combination of 
outside guidance and local knowledge increased a sense 
of ownership and acceptance among program staff. 
Assessment, in the words of one participant, was some¬ 
thing that QIS did “with providers instead of doing to 
them” (Spielberger & Lockaby, 2007, p. 23). Having staff 
participate in the assessment process made it easier for 
some to “open their minds,” accept the need for improve¬ 
ment, and focus their attention on specific areas of need 
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(Spielberger et al., 2009, p. 16). On occasion, program 
self-assessments were more critical than external assess¬ 
ments, a result that further diminished the threat from 
the external assessment. 

Even with this relatively low-stakes approach, how¬ 
ever, some program directors expressed trepidation at 
being assessed by outsiders. These concerns were of sev¬ 
eral types, including a worry that the assessment had 
taken place when the best staff weren’t present; that “mit¬ 
igating factors” were not appreciated, understood or tak¬ 
en into consideration; or that the baseline scores were 
not fully explained or understood (Spielberger et al., 
2009, p. 14). These concerns hint at the challenges of 
engaging programs even in lower-stakes appraisals of 
performance, underscoring the value of a supportive ap¬ 
proach like the one embodied in 
QIS for engaging programs in hon¬ 
est discussions about quality. 

Long-term and Continuous 
Quality Improvement 

Another reason for the effective¬ 
ness of the QIS was that it pre¬ 
sumed a long-term relationship 
with participating agencies and a 
continuous focus on quality im¬ 
provement. As programs entered 
the QIS during its phased rollout 
between 2006 and 2008, some 
program directors new to the QIS were generally skepti¬ 
cal of the low-stakes philosophy and wondered how pro¬ 
grams would be motivated to change without explicit 
external incentives. Longitudinal interviews, however, 
revealed that the year-long QIS cycle of assessment, plan¬ 
ning, and support kindled in program directors an in¬ 
trinsic interest in increasing quality. The process largely 
eliminated doubts about the strategy of using ongoing 
support, rather than specific rewards or sanctions, to en¬ 
courage the development of quality. The low-stakes ap¬ 
proach and the long-term nature of involvement were 
compatible strategies. 

Our interviews indicated that, among those direc¬ 
tors with some initial uncertainty about QIS, most found 
their concerns substantially reduced in just one year of 
participation. By the third year, the views of program di¬ 
rectors had converged on a high level of satisfaction with 
QIS and a belief in its positive effect on quality—whether 
the program directors had joined QIS enthusiastically 
and well-informed or had started with indifference and 
uncertainty or a more superficial understanding. Time 


and the ongoing QIS cycle worked together to allay con¬ 
cerns and allow programs to make changes over multiple 
years that may have been difficult for some program 
directors to imagine in the short term. 

Quality A dvisors 

As noted earlier, programs participating in the QIS are 
each assigned a quality advisor who reviews the findings 
from the external and self-assessments and helps to gen¬ 
erate an individualized program improvement plan. All 
the program directors interviewed expressed appreciation 
for their quality advisors and identified them as providing 
critical support. That support was concrete and practical, 
as quality advisors helped interpret assessment findings; 
conceptualize what improvements might look like in that 
specific program; and link pro¬ 
grams to training, curricular re¬ 
sources, and other supports pro¬ 
vided by Prime Time or partner 
agencies. The individualized nature 
of this process resulted in targeted 
use of resources, linking specific 
program weaknesses with specific 
resources such as trainings, “as op¬ 
posed to blindly sending your staff 
to all kinds of trainings,” as one 
program director characterized it 
(Spielberger et al., 2009, p. 14). 
The support from quality ad¬ 
visors was also relational. Quality advisors are one im¬ 
portant reason many program directors described the 
move from the QRS to the QIS as a shift from something 
directive to something that felt like “coaching” (Spielberger 
& Lockaby, 2007). Quality advisors were frequently de¬ 
scribed as trusted, reliable, flexible, and responsive; they 
developed long-standing relationships with programs 
and often provided social and emotional support. Quality 
advisors were described as encouraging program direc¬ 
tors to take ownership of the assessment process— 
choosing, for example, whether to conduct the self- 
assessment before or after drafting the improvement 
plan. They served as important advocates with Prime 
Time regarding program managers’ experiences with the 
QIS process and provided helpful links to and perspec¬ 
tives on developments outside the particular agency 
(Spielberger et al., 2009). Notably, program directors 
who were part of the QIS from its pilot phase and de¬ 
scribed particularly close relationships with their quality 
advisor also participated in large numbers of Prime Time 
services and supports (Spielberger et al., 2009). 


Quality advisors were 
frequently described as 
trusted, reliable, flexible, 
and responsive; they 
developed long-standing 
relationships with 
programs and often 
provided social and 
emotional support. 


42 Afte rschool Matters 


Fall 2010 


A System of Supports and Resources 

The low-stakes approach, the long-term and continuous 
nature of the QIS, and the engagement of quality advi¬ 
sors all serve as pathways to critically needed and tai¬ 
lored supports and resources. Accordingly, the QIS has 
also been effective because Prime Time built a system of 
supports, with each part playing a different role. These 
supports include a wide variety of educational, training, 
and career advising services offered to individual after¬ 
school practitioners, or at times to all staff at a specific 
afterschool program, through Prime Time’s professional 
development department. The community partnerships 
department provided additional resources. 

With a vision of a system of supports as the guid¬ 
ing concept, Prime Time’s profes¬ 
sional development team re¬ 
sponded to the learning needs of 
the community of practitioners in 
Palm Beach County, even as it at¬ 
tended to standards from the larger 
field. It worked to align the types 
and number of trainings with the 
goals afterschool programs identi¬ 
fied in their improvement plans. 

It guided training participants 
through the process of creating 
customized, practicable plans for 
their own afterschool programs. 

Trainings were explicitly linked to 
the five quality standards, the PBC-PQA tool, and a set 
of core competencies that Prime Time’s professional 
development team developed in coordination with 
other local partners. 

Prime Time’s community partnership team offered 
resources and services to eligible programs in Palm 
Beach County, with priority given to those in QIS. The 
community partnership team managed contracts with 
several local nonprofits to deliver a variety of enriching 
curricular enhancement activities. Afterschool programs 
could request high-quality activities for their youth in 
content areas such as arts and culture, sports, health and 
wellness, media arts, literacy, and science and technolo¬ 
gy. In addition to providing direct services to youth, 
these “enhancement” agencies were also expected to 
work with the afterschool staff to help them strengthen 
their skills in these content areas. These contracts helped 
afterschool programs offer challenging experiences— 
one of the five local afterschool quality standards—while 
simultaneously supporting future capacity of the after¬ 
school programs (Baker et al., 2010). 


As Prime Time increased the number of supports 
available to afterschool organizations, it fine-tuned what 
each part of the system provided to fill in gaps and re¬ 
duce overlap. Although it was not restricted to QIS par¬ 
ticipants, this larger system of supports and resources 
became an integral part of helping programs meet the 
goals they set for themselves and improve their quality 
one aspect at a time. 

Making Quality Work 

This article summarizes research that demonstrates the 
effectiveness of the Prime Time Quality Improvement 
System and highlights essential elements that appear to 
contribute to the system’s effectiveness. Perhaps of ut¬ 
most importance has been the con¬ 
ception of QIS as a long-term pro¬ 
cess with embedded supports. This 
approach has allowed Prime Time 
administrators to revisit and revise 
the QIS in incremental steps, first 
through the QRS that helped to in¬ 
form QIS development, through an 
18-month QIS pilot, and through a 
rollout to a larger and more diverse 
population of afterschool sites. The 
longevity and stability of the sys¬ 
tem has reassured program direc¬ 
tors who were able to increase their 
trust and involvement with QIS as 
they repeated the cycle of assessment, planning, sup¬ 
ports, and reassessment. The QIS allowed even programs 
at relatively low initial levels of quality to begin paying 
attention to improvement, with the expectation that a 
culture of quality could be nurtured whatever their start¬ 
ing point. More broadly, it has allowed the slow but 
steady growth of a learning community in Palm Beach 
County that values and aspires to high-quality afterschool 
programming. 

The Prime Time QIS provides a concrete example 
for afterschool programs outside Palm Beach County. As 
other research has demonstrated, even lower-quality pro¬ 
grams often have some strengths—for example, in ensur¬ 
ing participants’ physical health and safety—on which to 
build toward more difficult but important aspects of pro¬ 
gram quality such as youth engagement and youth-adult 
interactions (Sugar et al., 2009). As the afterschool field 
seeks to make the transition from focusing primarily on 
child safety to emphasizing enrichment and develop¬ 
ment, it has a strong interest in moving lower-quality 
programs toward these higher-order skills. The QIS has 


The QIS allowed even 
programs at relatively low 
initial levels of quality to 
begin paying attention to 
improvement, with the 
expectation that a culture 
of quality could be 
nurtured whatever their 
starting point. 
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demonstrated that these improvements are possible, that 
a supportive and long-term approach can help overcome 
program staffs’ uncertainty or fear about making these 
improvements, and that supports and services can be tar¬ 
geted and refined in a systematic fashion. 

As it continues to oversee QIS, Prime Time confronts 
challenges that are endemic in the field, including staff 
turnover, competing priorities, changes in funding re¬ 
quirements, and a need for more, and more diverse, 
funding. But a commitment to high quality, supported 
systematically over the long term, holds the promise of 
meeting these challenges and sustaining the improve¬ 
ments important to individual practitioners and the af¬ 
terschool field as a whole. 
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